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i. 


COMPENSATION OF POLARIZATION MODE DISPERSION IN SINGLE MODE 
FIBER FOR MAXIMUM-LIKELIHOOD SEQUENCE ESTIMATION 


BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention relates to detection of data in a communications system, and, more 
particularly, to compensation for signal dispersion in an optical fiber. 

Cross-Reference to Related Applications 

This application claims the benefit of the filing date of U.S. provisional application no. 
60/280,326, filed on 03/30/01 as attorney docket no. Bessios 3PROV. 

Description of the Related Art 

In many digital communications systems, a user generates digital information that is then processed 
into an encoded (e.g., error-correction encoded) and/or packetized stream of data. The stream of data is 
then divided into discrete blocks. Each of the blocks may be mapped onto a corresponding one of a 
sequence of code or symbol values ("symbols") chosen from a pre-defined alphabet A, and generated with 
a period Tg, sometimes referred to as the "baud" rate. For optical transmission of the digital information, 
an optical carrier operating with a wavelength of, for example, 1 3 1 0 nm or 1 550 nm, is modulated with the 
encoded stream of data. The modulated optical carrier is then passed through an optical fiber, such as a 
single mode fiber (SMF) having its lowest order bound propagating at 1310 nm or 1550 nm. 

The modulated optical signal transmitted through the optical fiber channel comprises a series of 
light pulses. Since a transmission medium may be modeled as a filter having a corresponding (frequency- 
domain) transfer fiinction and (time-domain) impulse response, the pulse transmitted through the channel 
may have its shape modified based on this transfer ftmction. The analog pulse shape may be modified in 
amplitude and phase, and also experience dispersion of the pulse. Consequently, the time duration of the 
pulse transmitted through a medium may extend over a period of time greater than the time duration of a 
particular symbol. Adjacent pulses transmitted through the medium may thus corrupt each other, which 
corruption is known as inter-symbol interference (ISI). This characteristic of the optical fiber (channel) is 
termed memory (e.g., if one adjacent pulse contributes to ISI, the memory "length" is one). 

roS#123920 (992 1024) j Case Name Bessios 3 


As bit rates in optical communication systems increase for high speed data transmission, such as 
rates above 10 Gbps, receivers are increasingly faced with mitigation of effects of pulse dispersion and ISI 
to the optical signal passing through the optical fiber channel. A receiver typically includes a detector 
formmg decisions for received, sampled channel output data ('^receive signal") corresponding to the 
transmitted pulses. These detectors may apply compensation/equalization to the receive signal, and 
employ algorithms such as maximum-likelihood sequence detection to reconstruct the sequence of pulses 
in the transmitted, encoded stream of data 

For optical communication systems, there are several sources of pulse dispersion through an SMF. 
One source of dispersion is chromatic dispersion that causes time-domain pulse broadening due to the 
different traveling velocities of each of the optical pulse's spectral components. Prior art methods of 
compensation for chromatic dispersion use an opposite dispersion-compensating fiber (DCF) that has a 
greater dispersion parameter, usually by a factor of 10. 

Another source of dispersion is polarization mode dispersion (PMD) that arises from imperfections 
in the circular symmetry of the fiber core. The imperfections typically are caused by manufacturing 
imperfections in the core, chemical impurities, and excessive bending or strain during installation. 
Imperfect circular symmetry results in birefringent SMF that causes two orthogonal principal polarization 
states (PPS) to propagate with different velocity through the fiber core. The resulting average differential 
group delay (DGD) is proportional to the square-root of the transmission distance. For example, an SMF 

having PMD of 10p5/>/Z , where L is the distance in kilometers, has a DGD of 100/?5 at a distance of 
100km. A distances greater than 100km and bit rates of greater than 10 Gbps, the DGD becomes 
significant when compared to the symbol period (7^=1 00p5 at 1 0 Gbps). For a 1 0 Gbps transmission data 

rate, the magnitude of the maximum allowable value for the mean DGD (r^) between two orthogonal 
PPSs has an upper bound of \QOps (i.e., (r^) < lOO/?^). The (SMF) channel may be modeled as a two- 
symbol dispersive channel with impulse response h(t\ and the model for a 10 Gbps channel that reflects 
statistical differential delay between the two PPS components may be as given in equation (1): 


with 8(-) defined as the delta function, (oc/(l-a)) defined as the power distribution ratio among the two 
orthogonal PPSs with 0 < a < 1 , and where Zd follows a Maxwell distribution as given in equation (2): 



(1) 


Vd)- 2/ \: 


jcxp - 



0< r^<oo 


(2) 
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ISI results in multi-level channel output values due to the efifect of the channel's memory on the 
binary input levels, causmg degradation of a receiver's bit error rate (BER) performance. Linear ISI and 
time-varying PMD effects are generally compensated for using adaptive equalization. 

SUMMARY OF THE INVENTION 

The present invention relates to spectrally shaping an output signal of a single mode fiber (SMF) 
to achieve characteristics of a predefined charmel 'target" response. The target response is that of a 
partial-response channel with additive white Gaussian noise. A receiver may then employ a maximum- 
likelihood sequence estimation (MLSE) detector having its detection algorithm, such as a Viterbi algorithm 
(VA), matched to the target response. Thus, state, branch, and path metric calculations for a Viterbi trellis 
may be optimized for a channel having this target response. 

In accordance with one embodiment of the present invention, compensation is applied to samples 
received from an optical channel by spectrally shaping, with an equalizer, the samples for compensation to 
generate a sequence of equalized samples. An error is generated for a current sample based on the 
difference between 1) an equalized current sample and 2) a decision for the current sample adjusted for a 
target response, wherein the target response is based on a response of the optical channel. The error is 
combined with one or more samples to provide an update signal; and the equalizer is updated with the 
update signal to adjust the equalizer response to the target response. 

BRIEF DESCRIPTION OF THE DRAWINGS 

Other aspects, features, and advantages of the present invention will become more fully apparent 
from the following detailed description, the appended claims, and the accompanying drawings in which: 

FIG. 1 shows a block diagram of a transmission system including a spectral-shaping detector 
operating in accordance with an exemplary embodiment of the present invention; 

FIG. 2 shows an exemplary 2-state transition diagram for the 2-state VA algorithm with branch 
metrics based on the parameter^ for transitions; 

FIG. 3 shows a block diagram of the spectral-shaping detector of FIG. 2; and 

FIG. 4 shows a block diagram of the finite impulse response (FIR) filter of FIG. 3. 
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DETAmED DESCRIPTION 

In accordance with exemplary embodiments of the present invention, an output signal of a single 
mode fiber (SMF) is spectrally shaped to achieve characteristics of a predefined channel ("target") 
response. The target response is that of a partial-response channel with additive white Gaussian noise 
(AWGN) (e.g., controlled inter-symbol intereference (ISI) is introduced). A receiver employs a maximum- 
likelihood sequence estimation (MLSE) detector having its detection algorithm, such as a Viterbi algorithm 
(VA), matched to the target response. Thus, state, branch, and path metric calculations for a Viterbi trellis 
may be optimized for a channel havmg this target response. A receiver employing one or more 
embodiments of the present invention may provide unproved bit error rate (BER) performance for 
detecting and receiving data from a carrier-modulated wavelength from a SMF. 

FIG. 1 shows a block diagram of a transmission system including a spectral-shaping detector 205 
operating in accordance with an exemplary embodiment of the present invention. Data d is transmitted 
through the channel comprising SMF 201 to which white, Gaussian noise is added, as shown conceptually 
as being added to the signal by adder 202. The signal from the channel is applied to matched filter 203 to 
reconstruct the transmitted pulses including the channel effects. The output of matched filter 203 is 
digitized into input samples with sampling frequency 1/T. The input samples are then applied to spectral- 
shaping MLSE detector 205 operating in accordance with an exemplary embodiment of the present 
invention. 

As described herein, an SMF with data transmission operating at 10-Gbps with (r^) < \QOps is 

characterized by a partial response with memory length one. Data generated for differential group delays 
of 70 ps and 1 00 ps and a given sampling phase may typically be transformed by the channel into data 
dominated by ISI of the type a+bD where a = 1, b--0.3. From equation (1), the SMF channel response 
H{p) has the form of a+bD, where D is the delay operator (the channel dispersion length is two symbols). 

Thus, data may be recovered from input samples from an SMF channel response HiD) of memory 
length one using a 2-state VA algorithm. The 2-state Viterbi detector is sufficient to de-convolve the ISI 
for optimum sequence estimation in the presence of AWGN. SNR loss from ISI ( SNRj^^ ) may be 

completely recovered by MLSE detection via a 2-state VA algorithm given normalized minimum 
(Euclidean) distance for a dispersive channel (memory length I), as in equation (3): 
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The present invention is not limited to these data transmission rates, memory lengths, and SMF channel 
response forms. In addition, more complex forms of target response may be selected. 

For the preferred embodiments, the target response T{p) is selected as + D, where ^ is a 
parameter that for some embodiments may be fixed. However, preferred embodiments of the present 
invention adaptively select A based on input samples over time, where A may vary between 0 and 1 . The 
parameter^ is incorporated into the branch metric of the two state VA algorithm. FIG. 2 shows an 
exemplary 2-state transition diagram for the 2-state VA algorithm with branch metrics based on the 
parameter^ for transitions. As shown in FIG. 2, the two symbol states are "1" and "-1". The branch 
metric for staying in state "1" is 1/(1+A) and for transitioning to state "-1" is -1/(A-1)- The branch metric 
for staying in state "-1" is -l/(-l-A) and for transitioning to state "1" is 1/{1-A). 

FIG. 3 shows a block diagram of the spectral-shaping, MLSE (SS-MLSE) detector 205 of FIG. 2. 
SS-MLSE detector 205 comprises SS finite impulse response (SS-FIR) filter 301, MLSE detector 302, 
threshold detector 303, response filter 304, combiner 305, multiplier 306, and squared-error accumulator 
307. 

In a first path, input sample Xk at time k is applied to SS-FIR filter 301 . As employed herein, the 
terms x{k) and Xk are equivalent notation for the sample at time k, SS-FIR filter 301 applies equalization to 
the sample xu to generate the output value yu. Equalization of SS-FIR filter 301 adjusts the input samples 
to correspond to samples passing through a channel with the target response A+D. Filter taps of SS-FIR 
filter 301 are set based on an update signal Uk generated by multiplier 306 in the second path, as described 
subsequently. The output 7^ of SS-FIR filter 301 at time k may be given as in equation (4): 


where c^P is the f tap coefficient of FIR filter 301 at time k. The equalized output samples yk are then 

applied to MLSE detector 302 which detects the transmitted data symbols based on the 2-state VA 
algorithm employing the 2-state transition diagram of FIG. 2. 

Depending on the (converged) sampling phase tu, different impulse response coefficients hk 
characterize the "sampled channel." Therefore, since the sampled channel shows spectral variation as a 
function of sampling phase r, both equalization and timing recovery loops may be jointly adapted via a 
common error signal e. The error signal may be derived as known in the art based on a cost function, such 
as the mean squared error, and the error signal is derived as the stochastic gradient (or approximation 



(4) 
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thereof) of the cost function, such as the least means squares (LMS) error. Preferred embodiments of the 
present invention employ a LMS error term for the common error signal e. 

Returning to FIG. 3, in a second path input sample Xk at time k is applied to threshold detector 303 
which generates a decision corresponding to the input sample X]^, During initialization (described 
subsequently), switch 320 is in position A to provide the decision of threshold detector 303 to response 
filter 301 . During steady state operation, switch 320 is m position B to provide the decision of MLSE 
detector 302 to response filter 301 . Either decision from switch 320 is applied to response filter 301 to 
generate an approximation to the data passing through a channel with the target response A+D. Combiner 
305 generates the error signal ifi^^ at time k as the difference between the output jjt of (equalizer) SS-FIR 
filter 301 and the desired target response signal Zk generated by response filter 304. 

The common cost function employed is the quadratic error , which is the highest variance 
unbiased estimate of the mean-squared error ^^(^f ) . As shown in Table 1, this cost function is minimized 
with respect to either sampling phase for timing recovery and FIR filter tap &^ for equalization. Table 
1 also gives the recursive update equation for each cost function. The recursive update equations are an 
approxunation for updated sampling phase % and FIR filter taps c^^ . 


Table 1 


Loop 

Optimization Cost Function 

Update Equation 

Sampling Phase 



(LMS) Filter Taps {{x is a 
predetermined constant) 


4\=4'-HekXk 


The constant // controls the rate of adaptation, and for preferred embodiments the constant //may be 
selected between 0<ju< {lIKizd^ where A^ax is the largest eigenvalue of the autocorrelation matrix of a 
filter-tap input vector. The error signal Ck is multiplied by the constant // and the current sample in 
multiplier 306 to generate the update signal m^. 

FIG. 4 shows a block diagram of an implementation of SS-FIR filter 301 of FIG. 3 using a 3-tap 
adaptive FIR filter. One skilled in the art would realize that the present invention is not limited to 3 taps, 
and that a given implementation of SS-FIR filter 301 may include more or less taps. The 3-tap FIR filter 
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employs LMS adaptation with recursive updates as given in Table 1 when operating as an equalizer. SS- 
FIR filter 301 comprises delays 401 and 402 generating Xk.\ and Xk.2 from Xk^ respectively. Multiplier 306 
of FIG. 3 is implemented by four multipliers 306(a)-306(d). Multiplier 306(d) combines the error with 
the constant and multipliers 306(a)-306(c) multiply Xk, Xk^u and Xk-i^ respectively, with //e^ 

5 Delay 403 and combiner 406 operate to update the FIR filter tap c^^^ with the output of multiplier 

306(a) with recursive update of Table 1. FIR filter tap c^^j is similarly updated by delay 404 and 
combiner 407, and FIR filter tap c^i^ similarly updated by delay 405 and combiner 408. FIR filter tap 
^ and current sample x^ are multiplied in multiplier 410, FIR filter tap cj[?J and previous sample Xk-i are 
multiplied in multiplier 411, and FIR filter tap c^jij and sample Xk-i are multiplied in multiplier 412. The 

1 0 output values of multipliers 4 1 0, 4 1 1 , and 4 1 2 are added together in combiners 4 1 3 and 4 1 4 to generate 

p i the current filtered sample yk. 

r. Since the sampled channel is spectrally close to the target response, the equalizer's noise 

HI enhancement and tap length N may be reduced. For the preferred embodiment, the precision (tap width) 

;,fi 

i|1 for filter tap values is 1 5 bits. The LMS adaptation takes place for every symbol (fiill baud rate) the rate 

5 for such recursive adaptation may be reduced by performing adaptation for sequential blocks of symbols 

5\ (e.g. every 20 symbols). The adaptation rate may depend on the rate of channel variation. 

yi Returning to FIG. 3, during startup or other initialization, the parameter A may be initially set 

pi with an adaptation process as follows. Squared-error accumulator 307 generates the current mean squared 

error (MSE) by accumulating for each k. The current MSE is a quantity related to the channel quality. 

20 Parameter A may be defined with a discrete set of levels, such as the set A={0.2, 0.4, 0.6, 0.8,1}. The 
current MSE value may then be employed to select a value for A from the discrete set of levels. The 
parameter A is adapted using an adjustment step of, for example, 0.2. Thus, the values for tap coefficients 
of FIR filter 301 are set to an initial value. Then the value of current MSE may be compared to an 
adaptation threshold, and while the value of current MSE is greater than the adaptation threshold, the 

25 parameter A is incremented by the adaptation step (when starting from a low initial value for A). When 
the current MSE is less than the adaptation threshold, the value for A is fixed, and taps of FIR filter 301 
are then adapted. Alternatively, f starting from a high initial value for A, the adaptation process may 
decrement by steps until the current MSE is less than the adaptation threshold. During initialization, the 
branch metrics of the 2-state VA such as shown in FIG, 2 may also be updated, or may be fixed until the 

30 end of the initialization process, at which time the value of A for the 2-state VA is fixed. 
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During operation, some embodiments may also adaptively update the value of A during steady- 
state operation by monitoring the value of the current MSE. For this case, if the value of MSE becomes 
greater than the adaptation threshold, then the parameter A is incremented by the adaptation step until the 
current MSE is less than the adaptation threshold. To reduce A, the parameter A is periodically 
decremented by an adaptation step and the current MSE compared with the adaptation threshold. If the 
current MSE is greater than the adaptation threshold, the value for A is incremented by the adaptation step, 
but, if the current MSE remains less than the adaptation threshold, the value for A is decremented again by 
the adaptation step until the current MSE is greater than the adaptation threshold, at which point the 
parameter for A is incremented by the adaptation step and set. Again, the branch metrics of the 2-state VA 
such as shown in FIG. 2 may also be updated during this process. 

While the exemplary embodiments of the present invention have been described with respect to 
processes of circuits, the present invention is not so limited. As would be apparent to one skilled in the art, 
various functions of circuit elements may also be implemented in the digital domain as processing steps in 
a software program. Such software may be employed in, for example, a digital signal processor, micro- 
controller or general purpose computer. 

The present invention can be embodied in the form of methods and apparatuses for practicing 
those methods. The present invention can also be embodied in the form of program code embodied in 
tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage 
medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, 
the machine becomes an apparatus for practicing the invention. The present invention can also be 
embodied in the form of program code, for example, whether stored in a storage medium, loaded into 
and/or executed by a machine, or transmitted over some transmission medium, such as over electrical 
wiring or cabling, through fiber optics, or via electromagnetic radiation, wherein, when the program code 
is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for 
practicing the invention. When implemented on a general-purpose processor, the program code segments 
combine with the processor to provide a unique device that operates analogously to specific logic circuits. 

It will be further understood that various changes in the details, materials, and arrangements of the 
parts which have been described and illustrated in order to explain the nature of this mvention may be 
made by those skilled in the art without departing ftom the principle and scope of the invention as 
expressed in the following claims. 
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